Testing newly developed software is an essential part of the process of software product development. Software developers use a variety of techniques to test software for errors. Often the software is tested at a "beta" test site; that is, the software developer enlists the aid of outside users to test the new software. The users, often under a nondisclosure agreement, will use the new software and report on any errors found in the software. This is a long and difficult process because many person-years of testing are required to ensure that the new software has no errors. If only one or two beta test sites are used, the process consumes long periods of time because the small number of users are less likely to uncover errors than a large group of testers using the software in a variety of applications. As a result, software developers generally use a large number of beta test sites in order to reduce the time required for testing the software. Beta testing is a costly process because of the large number of test sites involved.
Other software developers utilize automatic software testing in order to reduce the cost and time for software testing. In a typical automatic software testing system, the software is run through a series of predetermined commands until an error is detected. The automated test system will generally halt upon detection of an error. This deterministic type of automated testing is still a costly and time-consuming process because complete testing of the software product requires that the automatic test system use every possible permutation of options on the software being tested in order to ensure that unusual combinations of program commands will not cause an error.
While automatic software testing has the advantage that it detects errors that would not be found using only a limited number of beta test sites, errors must be corrected one at a time as they are detected by the automatic test system. On the other hand, beta testing has the advantage that a large number of beta test sites may detect a number of errors that can be corrected in a single revision rather than correcting one error at a time. However, neither approach offers a cost effective way to detect a large number of errors in software under development.
Therefore it can be appreciated that there is a significant need for a system and method of automatic testing which will allow thorough testing of software at a reduced cost and in a limited period of time.